Skip to content

Add a backwards linear function to be used with the fused mla q up-proj - #3330

Open
chaseblock wants to merge 3 commits into
NVIDIA:mainfrom
chaseblock:lin_bwd
Open

Add a backwards linear function to be used with the fused mla q up-proj#3330
chaseblock wants to merge 3 commits into
NVIDIA:mainfrom
chaseblock:lin_bwd

Conversation

@chaseblock

@chaseblock chaseblock commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

Description

Please include a brief summary of the changes, relevant motivation and context.

Fixes # (issue)

Type of change

  • Documentation change (change only to the documentation, either a fix or a new content)
  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Infra/Build change
  • Code refactoring

Changes

Please list the changes introduced in this PR:

  • Change A
  • Change B

Checklist:

  • I have read and followed the contributing guidelines
  • The functionality is complete
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes

@chaseblock
Add a backwards linear function to be used with the fused mla q up-proj
ed22445 
Signed-off-by: Chase Block <cblock@nvidia.com>
@github-actions github-actions Bot added the community-contribution PRs from external contributor outside the core maintainers, representing community-driven work. label Aug 7, 2026
chaseblock added a commit to chaseblock/Megatron-LM that referenced this pull request Aug 7, 2026
@chaseblock
Eliminate duplicate code, offload bwd to TE for fused qproj.
4da1c6f 
Requires TE PR: NVIDIA/TransformerEngine#3330

Signed-off-by: Chase Block <cblock@nvidia.com>
@greptile-apps

greptile-apps Bot commented Aug 7, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Greptile Summary

The PR adds a reusable linear-backward entry point for fused operations and exposes it through the fused MLA Q up-projection helper.

  • Constructs LinearBwdArgs for quantized and non-quantized projection backward passes.
  • Supports tensor/sequence parallelism, deferred or fused weight-gradient accumulation, and optional gradient computation.
  • Delegates the fused MLA Q up-projection backward path to the new helper.

Confidence Score: 4/5

The PR should not merge until the outstanding bias-gradient contract is fixed for callers that enable bias.

The new wrapper allows use_bias=True, and _linear_backward computes the corresponding gradient, but the wrapper discards that result and provides no other path for a fused caller to update the bias.

Files Needing Attention: transformer_engine/pytorch/module/linear.py

Important Files Changed

Filename Overview
transformer_engine/pytorch/module/linear.py Adds the shared backward helper, but its optional bias-gradient result remains discarded.
transformer_engine/pytorch/attention/fused_mla_q_uproj.py Adds a thin classmethod that forwards fused MLA projection-backward inputs and keyword options to the shared helper.

Reviews (2): Last reviewed commit: "Remove redundant import in backward_line..." | Re-trigger Greptile

greptile-apps[bot]
greptile-apps Bot reviewed Aug 7, 2026
View reviewed changes
Comment thread transformer_engine/pytorch/module/linear.py
Comment on lines +872 to +873
wgrad, dgrad, _ = _linear_backward(bwd_args)
return dgrad, wgrad

@greptile-apps greptile-apps Bot Aug 7, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Bias gradient is discarded

When use_bias=True, _linear_backward computes a bias gradient, but this wrapper discards the third return value and exposes only dgrad and wgrad, preventing the caller from updating the bias parameter.

Knowledge Base Used: PyTorch Fused Modules (transformer_engine/pytorch/module)

@chaseblock
Remove redundant import in backward_linear
554d644 
Signed-off-by: Chase Block <cblock@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@greptile-apps greptile-apps[bot] greptile-apps[bot] left review comments
@cyanguwa cyanguwa Awaiting requested review from cyanguwa cyanguwa is a code owner
@ksivaman ksivaman Awaiting requested review from ksivaman ksivaman is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

community-contribution PRs from external contributor outside the core maintainers, representing community-driven work.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@chaseblock